ITS Calib: fix parallel processing for highly disordered data #14756
ITS Calib: fix parallel processing for highly disordered data #14756mconcas merged 4 commits intoAliceO2Group:devfrom
Conversation
|
REQUEST FOR PRODUCTION RELEASES: This will add The following labels are available |
|
I need to perform another set of tests |
|
@fprino now this PR is ready. |
|
We have one failing check. Could it be due to a glitch in the system? |
|
Yes I think so since before my last commit it was again failing. From the error message, it does not seem to be related to this PR itself. |
|
I did a dummy commit since I will travel in the next hours. Can I kindly ask you to force merge if the test will fail again? Thank you very much. |
3 participants
I noted that when I process data locally, with multiple processes, and tf are not well ordered from the file list in input, some operations in the calib workflow are not safe.
P2 not impacted.
Still this PR modifies quite many lines. A full revalidation has been carried out locally on all ITS calibration runs.
New feature: the percentage of successful extractions is not normalized to the total number of scanned rows. Before it was normalized to the total number of rows processed ("seen") by the wf which can be different in some cases.
More details:
OTHER Features:
--localto be used with data replay when partial data are available (e.g. long runs for which a TF dump was only done partially).finalize()is now also called in theendOfStream()that helps in case of partial data (e.g. digital scan) both locally and at P2.